Why bother with these? If something fails, I thought the application will save it into a place where it can act as a regression test.

This is indeed the case when we run individual fuzz tests. However, CI will only run the test cases added by f.Add without trying other random values generated from the corpus. Will having more corpus be beneficial for CI? If not then I will keep only one of them.

I see, so you want it to run at least a few times. We discussed with @vitsalis that we could add -fuzztime to the CI to run it for a limited amount of iterations.

Not sure what you guys think about doing something like f.Add(time.Now().UnixMilli()) so the tests run as unit tests are not always the same, and you aren't forced to include this arbitrary looking boilerplate all the time. It would ensure that it runs them at least once, and if they fail then hopefully it prints the troublesome seed as well.

The counter argument was that it makes builds non-deterministic. It can be annoying if random tests fail sometimes, not other times, but over a long period of time it would catch more bugs than relying on the same seeds. We could even do something like addRandomSeeds(f, 100) to add exactly 100 random seeds, which is what other frameworks are doing (ie. they achieve 100 passes by default for each test, rather than run forever like Go).

ScalaCheck has CLI flags for the numbers, so maybe we could also support some env vars to adjust defaults. Although I never needed that, I was happy with 100 unless it took really long to generate the data and I had to lower it on a per-test or per-suite basis, like here

we could add -fuzztime to the CI to run it for a limited amount of iterations.

Are you aware of any projects using such strategy? I feel that this requires a very powerful VM to run the CI. My CPU was super hot when executing fuzz tests and sometimes a fuzz test takes >10s to generate the initial corpus.

We could even do something like addRandomSeeds(f, 100) to add exactly 100 random seeds, which is what other frameworks are doing (ie. they achieve 100 passes by default for each test, rather than run forever like Go).

This is a good idea. Basically we limit the number of tries (rather than the running time) for fuzz tests. I will look into this approach and find a way to try it in the subsequent PRs. Hopefully the CI will be happy to do this.

Yes the QuickCheck family runs 100 tests by default and it's up to you to tweak the numbers. I also want tests to finish in reasonable time, max 10 seconds, so if I see that they take longer then I have to reign in how much data I generate.

Maybe I don't need 1000 blocks with up to 100 transactions in each just to test something simple, and maybe I let my tree generation do too much branching and my tree of 10 depth is exponentially wide. Generating random bytes can also get out of hand, like opaque payloads, we don't need them to be hundreds of kilobytes long.

So it's a good indication of where time is not spent well.

And you are right I'd rather know that a particular test passed 100 runs than that some VM spent overall 1 minute testing all fuzz tests and may or may not have covered them equally.

It may be worth creating a constructor method like NewQueuedMessage(txid, msg) and let it calculate the hash itself, so we're not able to generate invalid ones. Just an idea.

Okay but why would it change? There are no transactions, and even if there were the test doesn't say how long an epoch is.

Shouldn't the test generate random registrations and check that the validator set doesn't change during an epoch, but it does at the end?

That's a good idea! However this requires mocking MsgCreateValidator or even MsgWrappedCreateValidator that includes BLS stuff. I have added a TODO on this and will do it in the subsequent PRs.